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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  lest  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


This  report  on  guidelines  for  research  reporting  was  presented  by  the 
senior  author  at  a  symposium  on  "Problems  with  Meta-Analysis  of  Decision- 
Making  Research"  at  the  13th  Annual  Meeting  of  the  American  Institute  for 
Decision  Sciences  on  November  18,  1981,  in  Boston,  Massachusetts.  The  guide¬ 
lines  were  developed  by  the  authors  as  a  consequence  of  their  experiences 
in  conducting  a  meta-analysis  of  the  career  counseling  outcome  research 
published  during  1950-1980.  A  substantial  part  of  that  published  research 
overlaps  with  the  research  reported  in  ARI  Technical  Paper  316.  This  1978 
report,  "Outcome  Measures  for  Career  Counseling  Research,"  was  authored  by 
Laurel  W.  Oliver  under  Army  Project  2Q762717A766. 
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SUFFICIENCY  IN  THE  REPORTING  OF  RESEARCH  RESULTS:  SOME  GUIDELINES 


BRIEF 


Requirement: 

The  US  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI)  conducts  research  on  issues  of  interest  to  the  Army. 
Similar  research  may  be  accomplished  by  other  services.  Government 
agencies,  and  in  the  civilian  community.  Over  time,  a  substantial 
amount  of  research  in  the  same  area  may  accumulate.  The  problem  then 
becomes  one  of  integrating  those  research  results  in  a  way  that  is 
meaningful  for  Army  decision  makers.  The  meta-analysis  approach  used 
by  Glass  (1976)  and  his  colleagues  has  proved  useful  for  integrating 
the  results  of  many  studies. 


Procedure: 

The  authors  of  this  report  are  conducting  a  meta-analysis  of  the 
career  counseling  outcome  research  published  during  the  period  1950- 
1980.  They  have  found  that  the  data  needed  for  this  type  of  research 
integration  are  not  always  available  in  research  reports  or  published 
articles.  Accordingly,  the  authors  have  developed  some  guidelines  for 
research  reporting  which,  if  followed^  would  facilitate  research  inte¬ 
gration. 


Findings: 

Inadequacies  of  research  reporting  encountered  by  research  inte¬ 
grators  include:  failure  to  provide  needed  data,  selective  reporting 
of  data,  and  incomp] °te  descriptions  of  samples  or  interventions. 
Attempts  to  obtain  needed  information  from  authors  are  generally  un¬ 
successful,  especially  if  considerable  time  has  elapsed  since  the  re¬ 
search  was  conducted.  The  guidelines  the  authors  present  are:  (1) 
Report  means  and  standard  deviations  (or  correlation  coefficients), 
and  N's  for  all  groups;  (2)  Report  exact  levels  of  significance  (if 
available)  and  degrees  of  freedom  for  significance  tests;  (3)  Organize 
data  for  analysis  by  outcome  variable  and  do  not  conduct  separate  analy¬ 
ses  for  every  item  in  an  instrument;  (4)  Report  nonsignificant  as  well 
as  significant  findings  and  report  results  for  all  groups  included  in 
the  analysis;  (5)  Insure  accuracy  of  data.  For  the  purpose  of  using 
research  results  for  organizational  decision  making,  it  is  the  respon¬ 
sibility  of  the  research  integrator  to  portray  the  results  of  the  meta¬ 
analysis  in  a  realistic  manner. 


Utilization  of  Findings: 

The  purpose  of  this  report  is  to  encourage  researchers  to  report 
their  research  results  in  a  manner  that  will  make  their  results  more 
useful  in  a  meta-analysis.  Given  adequate  reporting  of  research  results 
and  meticulous  implementation  of  the  meta-analysis  approach,  decision  makers 
will  have  access  to  more  valid  information  on  which  to  base  their  decisions. 


viii 


CONTENTS 


_  1 

Page  | 


INTRODUCTION  . 

GUIDELINES  FOR  RESEARCHERS  REPORTING  RESULTS 

Commonly  Encountered  Problems  . 

Suggestions  for  Authors  (and  Editors) 

BEYOND  SUFFICIENT  REPORTING  OF  RESULTS  .  .  . 

Making  Use  of  Integrative  Findings  .  .  . 
Limitations  of  Meta-Analysis  . 

REFERENCES  . 


ix 


SUFFICIENCY  IN  THE  REPORTING  OF  RESEARCH  RESULTS:  SOME  GUIDELINES 

INTRODUCTION 


At  a  meeting  of  the  American  Psychological  Association  (APA)  a  little 
over  a  year  ago,  Frank  Schmidt  stated  that  “the  most  important  problem  in 
psychology  and  the  social  sciences  today  is  the  failure  to  produce  cumulative 
knowledge”  (Schmidt,  1980).  We  agree  with  Dr.  Schmidt.  For  decades,  we 
have  conducted  research  on  a  wide  variety  of  organizational  topics.  These 
have  included  personnel  selection,  leadership  and  management,  socio-technical 
systems,  and  many  others.  Some  of  these  topical  areas  are  directly  relevant 
to  the  decisions  organizations  must  make  in  allocating  and  developing  their 
resources.  In  many  instances,  however,  we  have  discovered  that  this 
plethora  of  research  leaves  us  in  a  confused  state.  For  example,  many 
authors  find  it  difficult  to  arrive  at  general  summary  statements  which 
capture  the  essence  of  the  results  and  make  these  results  useful  to  the 
reader.  Thus,  as  researchers  we  seem  to  have  produced  a  great  many  findings, 
but  we  have  usually  not  integrated  the  results  of  our  research  in  a  meaningful 
way. 


Reviewers  who  use  traditional  narrative  or  literary  approaches  to 
integrating  the  results  of  many  studies  have  based  their  conclusions  on 
their  personal  reading  of  a  set  of  studies  on  a  given  topic.  The  conclusions 
drawn  from  these  customary  reviews  are  necessarily  subjective  and  may  be  even 
more  so  if  the  reviewer  has  excluded  certain  studies  deemed  methodologically 
inferior  or  otherwise  inappropriate.  Some  reviewers  employ  the  more 
sophisticated  "vote -counting"  (or  "box-score")  approach  in  which  the 
results  of  each  study  are  sorted  into  positive  significant,  negative 
significant,  and  nonsignificant  categories  and  bases  his  or  her  conclusions 
on  the  resulting  tallies.  Although  more  systematic,  and  certainly  more 
revealing  when  dealing  with  a  considerable  number  of  studies,  the  vote¬ 
counting  approach  does  not  take  into  account  the  size  of  the  effects 
tabulated  and,  as  Hedges  and  Olkln  (1980)  have  shown,  may  still  result  in 
biased  estimates  of  outcomes. 

Traditional  methods  are  not  adequate  for  reviewing  the  results  of 
a  sizable  body  of  resarch.  In  integrating  the  findings  of  career  counseling 
outcome  research,  we  are  following  the  lead  of  Glass  (1976)  and  others  by 
employing  techniques  of  "meta-analysis.”  Meta -analysis  provides  a  means  of 
quantifying  and  combining  results  of  individual  studies.  The  unit  of 
analysis  is  a  standardized  mean  difference  known  as  the  "effect  size.”  As 
Smith  and  Glass  (1977)  and  Smith,  Glass,  and  Miller  (1980)  have  defined  it 
in  applying  meta -analysis  to  the  psychotherapy  research,  the  effect  size  is 
the  difference  between  the  means  of  the  experimental  and  the  control  groups 
divided  by  the  standard  deviation  of  the  control  groups  on  a  dependent 
variable.  That  is. 
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Thus  only  three  statistics  are  needed  to  calculate  an  effect  size:  mean 
of  the  experimental  group,  mean  of  the  control  group,  and  standard  deviation 
of  the  control  group.  We  naively  assumed  that  these  basic  data  must  surely 
be  reported  in  every  study.  Not  sot  As  others  have  found  before  us,  we 
discovered  a  surprising  number  of  studies  that  contained  data  which  were 
insufficient  for  the  application  of  the  usual  meta-analysis  techniques.1 
Our  efforts  to  obtain  missing  data  from  the  authors  reporting  the  research 
met  with  only  partial  success.  The  longer  the  lapse  of  time  since  the  study 
was  published,  the  more  difficult  it  was  to  secure  the  data  we  needed. 

Although  the  content  of  our  meta-analysis  is  of  most  Interest  to 
those  in  the  field  of  career  counseling,  the  problem  this  paper  addresses 
is  the  inadequacy  of  research  reporting.  Our  experiences  have  convinced  us 
that  there  should  be  certain  basic  requirements  for  reporting  research. 

With  the  proliferation  of  research  in  almost  every  area  of  investigation, 
we  believe  the  Integration  of  research  results  will  become  an  increasingly 
important  task.  Thus,  it  is  important  for  authors  to  report  their  research 
results  in  a  manner  that  will  facilitate  efforts  to  combine  the  results  of 
many  studies.  The  purpose  of  this  paper  is  to  offer  authors  guidelines 
which  we  believe  will  make  their  research  findings  directly  usable  in  any 
subsequent  meta-analysis. 


GUIDELINES  FOR  RESEARCHERS  REPORTING  RESULTS 


Commonly  Encountered  Problems 

A  number  of  problems  may  ensue  when  one  embarks  on  a  project  of 
retrieving  data  from  research  reports  in  order  to  aggregate  the  results. 
In  our  review  of  the  career  counseling  outcome  research  published  during 
the  period  1950-80  (Oliver  &  Spokane,  1981),  we  encountered  a  number  of 
difficulties.  These  problems  included  the  failure  to  provide  needed 
data, ^  selective  reporting  of  data,  errors,  incomplete  descriptions  of 
interventions  or  samples,  and  the  difficulty  of  obtaining  required  data 


1There  are  ways  in  which  effect  sizes  can  be  estimated  when  final  status 
means  and  standard  deviations  are  not  reported.  However,  these  are  estimates 
calculated  from  other  data  which  may  or  may  not  be  reported.  See  Glass, 
McGaw,  and  Smith  (1981)  and  Rosenthal  (1980)  for  details. 

^Some  of  the  articles  we  included  in  our  integrative  analysis  of  the 
career  counseling  outcome  literature  came  from  journals  whose  audience 
consists  primarily  of  counselor  practitioners.  Such  articles  sometimes 
report  results  as  "statistically  significant”  and  do  not  include  means, 
standard  deviations,  or  the  level  of  significance.  Our  position  is  that 
authors  of  this  type  of  article  could,  in  most  cases,  include  a  small 
supplementary  table  or,  at  the  least,  make  their  basic  data  available  to 
other  researchers. 
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from  authors.  The  statistic  most  frequently  omitted  was  the  standard 
deviation.  Occasionally,  authors  failed  to  report  JK's.  We  also  found  a 
number  of  obvious  errors  in  tables.  (We  do  not,  of  course,  know  how  many 
unobvious  errors  we  failed  to  detect.)  Sometimes  the  descriptions  of  the 
career  interventions  were  so  limited  that  it  was  difficult  to  determine 
what  kind  of  an  intervention  was  intended  (let  alone  replicate  such  an 
intervention  from  the  description).  Inadequate  descriptions  were  particularly 
troublesome  in  categorizing  control  conditions.  We  had  decided  to  classify 
control  groups  as  "controls"  only  if  they  had  received  no  career  Intervention. 
Some  control  groups  (especially  those  labeled  "placebo”  controls)  had  in 
fact  received  a  minimal  intervention.  And  it  was  sometimes  not  possible  to 
tell  from  the  author's  description  how  the  subjects  in  such  groups  had  been 
handled.  Where  feasible,  we  requested  (by  telephone  or  mail)  the  required 
information.  Such  efforts  were  generally  not  successful.  People  move, 
die,  throw  out  data,  or  cannot  find  data  they  believe  they  have  kept.  We 
have  concluded  that  there  is  no  adequate  substitute  for  a  complete  presentation 
in  the  research  report. 


Suggestions  for  Authors  (and  Editors) 

In  this  section,  we  present  some  suggestions  we  believe  would  render 
research  reports  more  usable  for  research  Integration.  By  research  reports, 
we  mean  journal  articles,  organizational  reports  of  research,  and  conference 
pape rs. 

1.  Basic  data.  At  the  very  least,  researchers  should  report  means 
and  standard  deviations  (or  correlation  coefficients  if  a  correlational 
analysis)  plus  tJ' s  for  all  groups.  While  it  would  also  be  helpful  to  have 
these  data  given  for  the  entire  sample,  we  realize  that  journal  space 
limitation  may  preclude  this  course  of  action.  However,  these  data  could 
easily  be  included  in  the  original  institutional  report  of  the  research  (if 
there  Is  one).  In  any  case,  they  should  be  obtainable  from  the  author  of 
the  article.  If  change  scores  are  used,^  the  pretest  and  posttest  data 
(means  and  standard  deviations)  should  be  reported  as  well.  If  analysis  of 
covariance  is  used,  the  correlation  between  the  covariates  and  the  dependent 
variables  should  be  reported.  It  is  also  desirable  for  authors  to  report 
unadjusted  posttest  means  and  standard  deviations  if  no  pretest  differences 
among  groups  were  found.  Editors  and  reviewers  who  wish  to  conserve  costly 
journal  space  should  be  careful  not  to  displace  these  basic  data. 

2.  Significance  tests.  In  cases  where  significance  tests  have  been 
made,  authors  should  report  J?  ratios  or  t^  values  (or  correlation  coefficients) 
with  their  exact  probability  levels  (not  g>  .05,”  but  £  ■  .13”),  if  available. 


^Because  of  the  problems  associated  with  change  (difference/gain)  scores, 
we  would  not  encourage  researchers  to  use  them.  Knapp  (1980)  has  discussed 
Che  unreliability  of  change  scores  in  counseling  research.  See  the  Cronbach 
and  Furby  (1970)  review  article  and  Harris'  (1963)  book  for  more  detailed 
discussions  of  measuring  "change.” 
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and  degrees  of  freedom.  It  was  our  experience  thac  when  no  significant 
differences  resulted,  authors  sometimes  neglected  to  report  those  non¬ 
significant  results  and  merely  settled  for  a  statement  of  "no  significant 
difference  was  found.” 

3.  Organization  of  data  for  analysis.  Another  dilemma  facing  the 
would-be  research  integrator  is  the  way  authors  organize  and  analyze  their 
data.  We  encountered  a  number  of  studies  in  which  an  analysis  was  conducted 
on  each  item  in  a  questionnaire  with  data  reported  only  for  individual 
items.  In  such  cases,  we  feel  that  items  should  be  grouped  by  type  of 
dependent  variable  and  data  also  reported  for  each  cluster  of  items.  All 
items  relating  to  career  information  seeking,  for  example,  should  be 
subsumed  under  that  variable.  Even  if  subscales  of  a  test  or  Inventory  are 
of  individual  interest,  a  total  score  should  generally  be  reported  for  the 
instrument,  where  appropriate. 

4.  Selective  reporting  of  data.  In  a  few  instances,  authors  chose 
to  report  only  part  of  the  data  generated  from  a  study.  Although  this 
practice  is  certainly  acceptable  when  a  study  is  too  large  to  report  in  its 
entirety,  failure  to  report  nonsignificant  findings  or  deletion  of  results 
for  certain  groups  makes  proper  research  aggregation  impossible.  We  believe 
authors  should  report  results  for  all  groups  on  all  dependent  variables. 

3.  Making  data  available.  As  we  noted  earlier,  published  articles 
may  be  directed  to  a  practitioner  audience  rather  than  to  researchers. 

Such  articles  often  do  not  contain  the  basic  data  needed  to  calculate 
effect  sizes.  If  it  is  clearly  inappropriate  or  space  limitations  do  not 
permit  the  inclusion  of  a  table  reporting  means,  standard  deviations,  and 
1 <’s  for  all  groups,  we  encourage  authors  to  make  these  basic  data  available 
to  researchers  in  some  alternate  fashion.  The  difficulty  of  the  "make 
available"  procedure  is  that  it  is  not  always  possible  to  contact  authors, 
especially  if  a  considerable  period  of  time  has  elapsed  since  the  publication 
of  the  article.  Accordingly,  we  urge  editors  to  permit  the  Inclusion  of 
such  a  summary  table,  perhaps  as  a  short  appendix.  For  most  research 
published  in  practitioner  journals,  these  tables  should  not  be  very  extensive. 
Oliver  and  Spokane  (1981)  found  that  70  percent  of  the  career  counseling 
outcome  studies  they  surveyed  employed  only  one  or  two  dependent  variables. 

6.  Accuracy  of  data.  From  time  to  time,  we  noticed  errors  in  articles. 
We  suggest  authors  completely  recheck  data  in  the  final  versions  of  their 
reports  against  the  computer  printouts  containing  their  results.  If  an 
article  is  revised,  the  final  revision  should  again  be  checked  against  the 
printout.  Page  proofs  can  then  be  checked  against  the  final  version.  (We 
of  course  believe  authors  should  check  and  recheck  their  entire  papers, 
but  here  we  are  focusing  on  the  data  required  for  meta-analysis). 


BEYOND  SUFFICIENT  REPORTING  OF  RESULTS 

We  discovered  a  number  of  shortcomings  in  the  published  research  in 
career  counseling  as  we  attempted  to  integrate  the  results  across  a  large 
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number  of  studies.  We  have  reported  these  deficiencies  in  the  hope  that 
future  reviewers  might  encounter  fewer  stumbling  blocks.  We  realize 
that  our  suggestions,  even  if  heeded,  will  not  be  reflected  in  the  content 
of  journal  articles  for  a  considerable  period  of  time.  An  even  longer 
period  would  be  required  for  the  findings  to  be  incorporated  into  an 
integrative  analysis.  Garvey  (1979)  has  suggested  that  the  process  of 
disseminating  and  communicating  findings,  from  the  time  a  given  study  is 
completed  until  it  appears  as  a  part  of  a  body  of  knowledge  in  a  textbook 
takes  13  years.  We  made  use  of  the  data  available  to  us,  however  limited, 
in  our  analysis  of  the  effects  of  career  counseling. 

We  do  want  to  comment  briefly  in  two  areas,  namely,  the  applications 
of  such  aggregate  findings  in  the  decision-making  process  and,  secondly, 
the  limitations  of  meta-analysis  that  argue  for  restricting  generalizations 
about  the  findings. 


Making  Use  of  Integrative  Findings 

Glass  and  Smith  (1980)  have  described  meta-analysis  as  "nothing  more 
than  the  attitude  of  data  analysis  applied  to  quantitative  summaries 
of  individual  experiments”  (Glass  and  Smith,  1980,  p.  277).  In  our 
experience,  however,  the  set  of  statistical  techniques  embodied  in  the 
meta-analysis  approach  are  employed  to  derive  summary  statements  with 
respect  to  a  body  of  research.  Because  these  techniques  are  increasing  in 
popularity  and,  in  light  of  some  cogent  criticisms  of  meta-analytic  procedures 
that  have  recently  been  proposed,  we  feel  that  it  is  important  to  delineate 
areas  in  which  these  findings  can  be'  usefully  employed.  Here,  we  rely 
substantially  on  points  made  by  Pillemer  and  Light  (1980). 

Translating  the  results  of  empirical  research  to  non-scientists. 

A  number  of  groups  may  have  interest  in  the  conclusions  drawn  from  lnter- 
grative  reviews.  In  the  field  of  career  counseling,  this  might  include  a 
large  and  diverse  group  of  professional  practitioners  of  varying  levels  of 
sophistication.  These  consumers  may  wish  to  extract  a  germ  from  the 
evidence  without  having  to  wade  through  a  profusion  of  individual  studies. 
Other  groups  may  Include  teachers  of  counseling,  directors  of  counseling 
centers,  the  general  public,  or  makers  of  public  policy  (e.g.,  legislators). 
Some  members  of  these  groups  have  a  tendency  to  seek  simple  answers  to 
complex  problems.  They  might  not  be  inclined  to  recognize  the  limitations 
Inherent  in  conclusions  from  integrative  reviews. 

Pillemer  and  Light  (1980)  have  suggested  that  meta-analysis  benefits 
the  reviewer  by:  (a)  Increasing  power  by  increasing  sample  size,  (b) 
obtaining  a  more  precise  estimate  of  effect  magnitudes,  (c)  describing  the 
form  of  a  relationship,  and  (d)  harnessing  the  results  of  contradictory 
findings.  These  advantages  improve  the  generalizability  of  reviews,  but 
must  be  carefully  qualified.  If  decisions  are  to  be  based  on  such  data, 
then  estimates  of  our  confidence  in  the  findings  should  be  provided  e.g., 
by  employing  Rosenthal's  (1980)  "file  drawer”  calculations  for  estimating 
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the  number  of  studies  with  null  results  not  included  that  would  be  necessary 
to  overturn  a  given  conclusion.  Meta-analytic  findings  are  temptingly 
clear  and  simple.  We  caution  that  there  are  rarely  neat  answers  to  complex 
questions  and  that  the  intricacies  of  these  procedures  are  difficult  to 
convey.  Nonetheless,  meta-analysis  does  seem  useful  in  decision  making. 

Information  about  the  efficacy  of  interventions.  Meta-analysis  has 
been  used  to  garner  data  on  the  differential  effectiveness  of  one  form  of 
treatment  or  intervention  as  compared  to  another.  This  has  been  done  with 
the  literature  on  psychotherapy  (Smith  and  Glass,  1977),  sex  bias  in 
counseling  (Smith,  1980),  and  instructional  practices  (Kulik,  Kullk,  & 

Cohen,  1979).  To  the  extent  that  such  treatments  are  comparable  with 
respec*  to  the  differential  selectivity^  of  subjects  they  employ,  as  well  as 
the  duration.  Intensity,  and  frequency  of  treatment,  such  comparisons  may 
well  be  useful.  However,  we  agree  with  Gallo  (1978)  who  suggests  that 
decisions  about  which  treatment  to  employ  eventually  rest  on  the  relative 
costs  to  the  individual  or  society  of  that  method.  A  small  improvement  in 
one  arena  may  be  considerably  more  valuable  and  important  than  a  large  one 
in  some  less  critical  area.  The  context  of  these  decisions  may  in  some 
cases  be  more  important  than  the  findings. 


Limitations  of  Meta-Analysis 

We  feel  that  the  principal  shortcoming  of  meta-analysis  is  the  false 
sense  of  security  that  use  of  the  approach  may  engender  in  the  user  of  the 
aggregated  research  results.  Quantifying  study  outcomes  and  applying 
statistical  techniques  to  the  resulting  effect  sizes  may  imply  a  greater 
degree  of  precision  than  actually  exists.  In  addition,  use  of  the  meta¬ 
analysis  approach  usually  results  in  general  summary  statements  which 
encapsulate  the  results  of  a  sizable  body  of  research.  Cook  and  Levlton 
(1980)  believe  meta-analysts  run  the  risk  of  overlooking  "the  Importance  of 
contingency -specifying  interactions  that  in  most  situations  have  an  infer¬ 
ential  precedence  over  statements  about  main  effects”  (p.  464)  and  give  an 
example.  While  we  agree  that  meta-analysts  may  ignore  interactions,  we 
feel  this  can  also  happen  with  other  methods  of  research  intergration. 

There  are  a  number  of  other  meta-analysis  limitations,  and  these  have  been 
ably  discussed  Jackson  (1980)  as  well  as  Cook  and  Levlton  (1980). 

For  the  purpose  of  using  research  results  for  organizational  decision 
making,  we  feel  it  is  the  responsibility  of  the  research  Integrator  to 
portray  the  results  of  the  meta-analysis  in  a  realistic  manner.  This 
presentation  of  results  obviously  must  follow  a  thorough  literature  search, 
a  thoughtful  choice  of  pertinent  study  characteristics,  reliable  coding, 
and  careful  analysis.  Given  adequate  reporting  of  research  results  and 
meticulous  implementation  of  the  meta-analysis  approach,  we  believe  decision 
makers  will  have  access  to  more  valid  information  on  which  to  base  their 
decisions. 
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